The operation of a conventional mechanical or optical pointing or input device such as a mechanical or optical computer mouse is well known in the art. By use of these devices, the user can select files, programs, or actions from lists, groups of icons, etc., and can “gesturally” move files, programs, etc. issue commands or map to specific actions, for example in drawing programs.
As examples, a mechanical computer mouse relies on one or more wheels and/or balls to track movement or displacement information relative to forward-backward and left-to-right movement of the computer mouse, for example by interrupting infrared beams of light directed at light sensors to create pulses representative of wheel or ball movement. Simple logic circuits interpret the relative timing of the pulses to indicate which direction the wheel(s) or ball(s) is moving, which is then converted by driver software into motion of a visual indicator such as a pointer, cursor, or cross-hair along X and Y axes of a computing device display screen.
An optical computer mouse replaces the mechanical mouse wheels or balls with one or more light sources such as light-emitting diodes (LEDs) or laser diodes to detect movement of the mouse relative to an underlying surface such as a mouse pad. The inertial/gyroscopic computer mouse uses a tuning fork or other accelerometer to detect rotary movement for every axis supported, most commonly using 2 degrees of rotational freedom and being insensitive to spatial translation. The user need only perform small wrist rotations to move a pointer or cursor on a display screen.
The underlying technology of modern surface-independent pointing/input devices such as optical mice (see FIGS. 1-2) is known as digital image correlation. The mouse includes a shell or housing 1 and a variety of input means such as left and right buttons 2, 3, a scroll wheel 4, etc. (see FIG. 1). A displacement detection LED 5 disposed on a bottom surface of the mouse (FIGS. 2-3) is used to detect movement of the mouse over a surface S (FIG. 3). An optical or optoelectronic mouse uses an optoelectronic sensor 6 (essentially, a tiny low-resolution video camera; see FIG. 3) to image a naturally occurring texture of a surface S made of materials such as wood, cloth, mouse pad materials and Formica, using light reflected from surface S via a focusing lens 7. These surfaces, when lit at a grazing angle by displacement detection LED 5 (see arrows in FIG. 3), cast distinct shadows that resemble a hilly terrain lit at sunset. Images of these surfaces are captured in continuous succession as the mouse is translated over the surface S, often at a speed of more than one thousand frames per second. Depending on the speed with which the mouse is moved, each image will be offset from the previous image by a fraction of a pixel or as many as several pixels. By using cross correlation to calculate how much each successive image is offset from the previous image, a displacement processor 8 can determine the distance the mouse has moved from image data captured by the sensor 6. The movement of the mouse can then be translated or converted into a corresponding movement of a visible marker such as a cursor on a graphical user interface such as a computer screen.
Even though a special surface S such as a mouse-pad is not needed by a modern optical mouse, a surface is still required for operation of the mouse. If a suitable operating surface S is not available and an alternative such as a touchpad or trackball is also not available, a conventional optical mouse cannot be used. In turn, certain tasks often done with pointing/input devices such as a computer mouse are difficult to impossible to accomplish with alternative pointing/input devices such as touchpads or trackballs. For example, use of drawing programs without a computer mouse can be difficult if not impossible. Likewise, tasks such as two-dimensional (2D) or 3D sculpturing or drawing, “flying” in multi-dimensional space (for example, three-dimensional space defined by X, Y, and Z axes) such as during gaming, etc. would be difficult to accomplish using a conventional touchpad, trackball, etc. For this reason, attempts have been made to adapt the familiar computer mouse to operate in the air or “on the fly,” to avoid the need for a surface S over which to translate the mouse for operation.
Two conventional approaches exist for operating a mouse or other pointing/input device in the air, i.e. without translating over an underlying surface S: an active approach and a passive approach. The approach taken depends on whether the mouse has a displacement detection system that works in 3D space. The optical displacement detection system of an optical mouse (see FIG. 3) can only work on a surface S due to the operating mechanism summarized above; it cannot work if suspended in 3D space.
In the active approach, typically an imager such as an IR camera is integrated into the pointing device to detect lights from an IR emitter of a console such as the console of a gaming device, and calculate spatial coordinates for the pointing device accordingly. The Wii® Remote marketed by Nintendo® falls within that category. A problem with this approach is that the pointing device spatial coordinates can only be calculated when its imager has a direct line of sight to a sensor bar associated with the gaming device console.
For pointing devices that do not include a distance measuring component, a passive approach has been evaluated requiring a separate component to measure the distance between the pointing device and, for example, a gaming device or base station, or to identify the location of the pointing device with respect to the gaming device or base station. All gesture-based pointing device approaches, such the Kinect® device marketed by Microsoft®, belong to this latter category. In this case, the fingers or the hands of a user play the role of a pointing device and a special imaging device is required to identify the locations of the fingers or hands of the user. Three-dimensional mice such as 3Dconnexion/Logitech's® SpaceMouse® in the early 1990s and Kantek's® 3D RingMouse® in the late 1990s, also known as bats, flying mice or wands, also fall in this category. As an example, the RingMouse® was tracked by a base station through ultrasound. This approach has been found to provide insufficient resolution.
Still other attempts have been made using one-dimensional imaging devices to track a point light source, to recognize and execute gestures input by the operator. The problem with this approach is that a holding time is required before each operation such as a click or drag can be performed.
To date, the present inventors are unaware of any attempts to operate a conventional optoelectronic computer mouse in the air directly using a conventional imager such as a webcam as an imaging device to support 2D or 3D pointing on any type of display screen, 2D or 3D.